Conventionally have been known and commercially available various types of apparatus for monitoring circumstance around a vehicle. One of the apparatus includes a camera mounted, for example, on a vehicle rear portion, and a monitor mounted near a driver's seat and for displaying a view captured by the camera as an image. Japanese Patent Laid-Open Publication No. 2001-187553 (hereinafter, referred to as a patent document 1) discloses a parking support system. The parking support system includes an image capturing unit, which captures different surroundings of a moving vehicle at first and second points as first and second images which are produced in time series, a stereoscopic object specifying unit, which detects and specifies a stereoscopic object in each first and second image, a vehicle position calculating unit, which calculates a vehicle moving data from the first point to the second point, and a stereoscopic object distance calculating unit, which calculates a distance between the vehicle and the stereoscopic object based upon positions of the stereoscopic object in the first and second images and the vehicle moving data. The parking support system including the above structure generates a third image to be transmitted to a vehicle driver based upon the images by the image capturing unit and the distance calculated by the stereoscopic object distance calculating unit. The image capturing unit is represented by a single camera. Therefore, a distance towards the stereoscopic object imaged in the first and second images can be calculated in accordance with principle of triangulation.
In an image processing technical field, has been known a conventional technique which reconstructs a three-dimensional shape by use of two cameras. For example, a nonpatent publication 1 (Koichiro Deguchi, ed., “2. Solve a Stereophonic Mechanism” in Information Processing, vol. 37, no. 7. Japan, 1996) describes gauging a stereoscopic object shape by a stereoscope method. A related portion in the nonpatent publication 1 holds followings as its prerequisites, to determine a position of a corresponding point in the space: features of the respective images shot by the two cameras, e.g., focal lengths of the two camera lenses, image centers thereof, pixel size thereof; positions and postures of the two cameras; and correspondence between the two images.
Likewise, a nonpatent publication 2 (Kenichi Kanatani, ed., “Three-Dimensional Shape Reconstruction by a Stereoscope Vision” in Mathematics of Spatial Data, vol. 1. Japan: Asakura Inc, 1995, 161-162) describes a method of calculating a three-dimensional shape of a substance in accordance with principle of triangulation based upon a corresponding relationship between the images shot by two cameras. The principal of triangulation has been described as one of the most fundamental methods of deriving three-dimensional information from images obtained by the two cameras, for the purpose of robot controlling.
Further, a nonpatent publication 3 (Koichiro Deguchi, ed., “3. Operation of Motion Images” in Information Processing, vol. 37, no. 8. Japan, 1996) describes that a three-dimensional shape of an object can be reconstructed based upon motion images in accordance with the same principle as the above-described stereoscope method. Especially, when the motion images are sequentially produced, the motion of the object can be tracked. Therefore, the sequential motion images are considered to be more preferable compared with two images in regard to deriving corresponding points.
Further, a nonpatent publication 4 (J. Borenstein, L. Feng, eds., “Gyrodometry: A New Method for Combining Data from Gyros Odometry in Mobile Robots” in Proceedings of the IEEE International Conference on Robotics and Automation. U.S., Apr. 22-28, 1996, 423-428) presents a very simple, yet very effective method of combining measurements from a gyro with measurements from wheel encoders (odometry).
Still further, a nonpatent publication 5 (Richard Hartley, Andrew Zisserman eds., “1.3 Projective Transformations” (p.11-) and “7.3 Action of a Projective Camera on Quadrics” (p.190-) in Multiple View Geometry in Computer Vision. U.K.: Cambridge University Press, 2000) presents that calibration between images at two different points, both of which include an identical object, can be represented by homography H.
A nonpatent publication 6 (Oliver Faugeras, ed., “3.4 Calibrating Cameras” in Three-Dimensional Computer Vision: A Geometric Viewpoint. U.S.: MIT Press, 1993, 51-58) presents that internal and external camera parameters can be calculated by analyzing homography matrix (perspective projection matrix).
The above-described patent document 1 discloses a method of calculating the distance toward the stereoscopic object in the first and second images. The parking support system is not required to have a gauging sensor, and yet is provided with the stereoscopic object specifying unit, which detects and specifies the stereoscopic object shot in the images, the vehicle position calculating unit, which calculates the vehicle moving data, and the stereoscopic object distance calculating unit, which calculates the distance between the vehicle and the stereoscopic object. More specifically, an image A shot while the vehicle 1 is at a position a is calibrated to an image as if it were shot while the vehicle 1 was at a position b, wherein an image C can be calculated. Therefore, a distance between the vehicle and the stereoscopic object can be geometrically calculated based upon a variation between the two images and the vehicle moving data.
Further, according to the above-described patent document 1, a steering angle sensor and a vehicle wheel speed sensor with a pulse counter are required to compute a vehicle moving distance and a vehicle moving direction. However, these sensors may raise following problems. First of all, the vehicle wheel speed sensor may not achieve sufficient detecting precision when the vehicle drives at a creeping speed. Therefore, output from right and left vehicle wheel speed sensors may differ due to imbalance of each wheel diameter. Further, an effective wheel diameter in response to load and an effective wheel base may not be determined precisely. Secondarily, there is a zone in which the steering angle sensor and turning radius are not nonlinearly related. Further, a vehicle movement is slow to respond relative to change of a steering wheel turning angle. Still further, the vehicle forward or rearward movements can not be recognized only with the steering angle sensor and the vehicle wheel speed sensor.
Still further, according to the above-described patent document 1, the turning radius is calculated based upon output from the steering angle sensor, and the pulse counter performs integration and counts the number of pulses outputted from the vehicle wheel sensor, by which the vehicle moving distance can be calculated. However, due to characteristics of the vehicle wheel sensor, integration error may occur or may be increased when the vehicle is more approaching a target position or when the target position is located more distant from a start point.
According to the above-described nonpatent publication 2, it is prerequisite that the two camera positions are fixed at left and right sides and the relative positions and postures are known. The distance toward the object can be calculated by identifying an identical point between images including the object shot by the left and right cameras in accordance with the principle of triangulation. This identification of the identical point between the images are referred to as a stereoscopic correspondence.
On the other hand, when the image processing technology described in the nonpatent publication 6 is referred, the three-dimensional shape of the object or the obstacle can be reconstructed only with a single camera. If this type of image processing technology is properly used by a system for observing surroundings of the movable body such as a vehicle, the above problems can be solved. However, according to a motion stereoscope method by the single camera, the camera position and posture varies in response to the vehicle movement. Accordingly, it requires a method of detecting both of the camera position and posture prior to the vehicle movement and the camera position and posture after the vehicle movement.
In regard to the method of detecting the camera position and posture, the nonpatent publication 4 discloses the method of combining measurements from a yaw rate sensor (a gyro) with measurements from a wheel speed sensor (an encoder). Further, the nonpatent publication 4 reminds that the error of the output from the yaw rate sensor is increased over time due to a time-dependent zero drift of the yaw rate sensor. This nonpatent publication 4 further reminds that it may be difficult to detect the vehicle position and posture because the vehicle wheel speed sensor generally tends to cause a relatively large error. In order to solve the problems, this nonpatent publication 4 suggests calculation of both error components such that the respective errors by the yaw rate sensors and the vehicle speed sensors are restrained so as not to practically affect on the vehicle position and posture detection.
According to the nonpatent publication 5, the calibration between the two images for the identical object is represented by homography H. A flat portion of each image has to contain at least four points in order to determine homography H. In other words, a relative positional relationship between a camera at first and second positions can be calculated if at least four points are contained in the flat portion of each image. That is, the other image can be calculated based upon one image. According to the nonpatent publication 6, the internal and external camera parameters can be calculated by analyzing homography H.
A need thus exists for providing an improved movable body circumstance monitoring apparatus which can high-precisely specify the position and posture of a single image capturing device, i.e., the movable body without causing errors. The improved movable body circumstance monitoring apparatus can assure a view from the movable body at a desired position and can properly display the image of the view.